An evaluation of course evaluations

Student ratings of teaching have been used, studied, and debated for almost a century. This article examines student ratings of teaching from a statistical perspective. The common practice of relying on averages of student teaching evaluation scores as the primary measure of teaching effectiveness for promotion and tenure decisions should be abandoned for substantive and statistical reasons: There is strong evidence that student responses to questions of “effectiveness” do not measure teaching effectiveness. Response rates and response variability matter. And comparing averages of categorical responses, even if the categories are represented by numbers, makes little sense. Student ratings of teaching are valuable when they ask the right questions, report response rates and score distributions, and are balanced by a variety of other sources and methods to evaluate teaching. Since 1975, course evaluations at University of California, Berkeley, have asked: Considering both the limitations and possibilities of the subject matter and course, how would you rate the overall teaching effectiveness of this instructor? 1 (not at all effective), 2, 3, 4 (moderately effective), 5, 6, 7 (extremely effective) Among faculty, student evaluations of teaching (SET) are a source of pride and satisfaction—and frustration and anxiety. High-stakes decisions including tenure and promotions rely on SET. Yet it is widely believed that they are primarily a popularity contest, that it is easy to “game” ratings, that good teachers get bad ratings and vice versa, and that rating anxiety stifles pedagogical innovation and encourages faculty to water down course content. What is the truth? We review statistical issues in analyzing and comparing SET scores, problems defining and measuring teaching effectiveness, and pernicious distortions that result from using SET scores as a proxy for teaching quality and effectiveness. We argue here—and the literature shows—that students are in a good position to evaluate some aspects of teaching, but SET are at best tenuously connected to teaching effectiveness (defining and measuring teaching effectiveness are knotty problems in themselves; we discuss this below). Other ways of evaluating teaching can be combined with student comments to produce a more reliable and meaningful composite. We make recommendations regarding the use of SET and discuss new policies implemented at University of California, Berkeley, in 2013.